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Effects of electron correlations on a two dimensional quantum spin Hall (QSH) system are studied. 
We examine possible phases of a generalized Hubbard model on a bilayer honeycomb lattice with 
a spin-orbit coupling and short range electron-electron repulsions at half filling, based on the slave 
rotor mean-field theory. Besides the conventional QSH phase and a broken-symmetry insulating 
phase, we find a new phase, a fractionalized quantum spin Hall phase, where the QSH effect arises 
for fractionalized spinons which carry only spin but not charge. Experimental manifestations of the 
exotic phase and effects of fluctuations beyond the saddle point approximation are also discussed. 



I. INTRODUCTION 

The quantum spin Hall (QSH) phase is a new state of 
matter which arises due to spin-orbit coupling in time- 
reversal symmetric systems 1 ^. It is characterized by a 
gap in the bulk and an odd number of Kramers pairs of 
gapless edge modes which are protected by a Z2 topologi- 
cal order-i* 2 * 3 -. Recently, an experimental signature for the 
gapless edge modes has been observed in HgTe quantum 
wells 4 -. Although the QSH state was proposed in a non- 
interacting system 1 , the gapless edge modes are stable 
in the presence of weak timc-rcversal symmetric disorder 
or many-body interactions^, which suggests that the 
topological order in the bulk is also robust against weak 
disorder— and interaction ;] 8 ' 9 ' 10 . If electron correlations 
become sufficiently strong, a broken-symmetry insulat- 
ing phase can be stabilized. Recently, a possibility of the 
QSH effect arising from many-body interactions has also 
been studie d 11 ' 12 . 

Fractionalized phases are another novel state of matter 
and arise as a result of electron correlations. In the ab- 
sence of spin-orbit coupling, a subtle balance between the 
kinetic energy of electrons and electron-electron interac- 
tions can stabilize spin liquid phases where spins remain 
disordered due to quantum fluctuations while charge ex- 
citations are gappe d 13 ' 14 . Spin liquids exhibit fraction- 
alization in that the low energy excitations are spinons 
which carry only spin but not charge. Much attention has 
been paid to two-dimensional frustrated magnets which 
are candidates for spin liquid states 1 ^. 

Since either spin-orbit couplings or electron correla- 
tions can lead to an interesting phase, what happens 
when both of these interactions are important? We ad- 
dress this question by examining the possibility of a new 
phase of matter arising due to an interplay between the 
spin-orbit coupling and electron correlations. A frac- 
tional QSH state which corresponds to a time-reversal 
symmetric version of the fractional quantum Hall state 
has been suggested as a possible phase for interacting sys- 
tems with spin-orbit coupling-. In this paper, we explore 
an alternative possibility where the QSH effect arises si- 
multaneously with fractionalization in a spin liquid state. 
The honeycomb lattice is an ideal geometry to study such 
effects because it may support both the QSH phase 1 - and 



the spin liquid phas o 16 ' . 



II. THE MODEL 

We consider a generalized Hubbard model defined on 
a double layer of honeycomb lattice, 

H = - E CM) (Uja*cl aiJ C jaa + h.C.) + U J2t,a ( n ™ ~ l f 

+U' Ei - 1) ("a - 1) ~ Eia V* («<» - 1) (!) 

where c iaa is the creation operator for an electron of spin 
a = ±1 on site i of layer a = 1 or 2, and iii a is the number 
operator. U(U ) is the on-site (interlayer) Coulomb re- 
pulsion and fi a is the chemical potential which is tuned so 
that each layer is at half filling. The intralayer tunneling 
amplitudes are tij a<T — t when are nearest neighbor 
(nn) sites and tijaa — S a %i ' e 1 ^" for next nearest neigh- 
bor (nnn) sites. We assume that there is no spin-orbit 
coupling or nnn hopping in the second layer and no inter- 
layer tunneling. The spin-dependent phase 4>ij(T, which 
we take to be positive (negative) if an electron with spin 
up (down) hops around the lattice in a counter clockwise 
sense, is due to spin orbit coupling^. We emphasize that 
our model is an idealized model and the goal of our inves- 
tigation is to demonstrate the possibility of finding a new 
state of matter from the simplest model which contains 
both spin-orbit coupling and electron correlations. 

We now represent the Hamiltonian in the slave rotor 
representation 1 ^, Ci aa = e~ l6ia fi a <j, where the spinon op- 
erator, fiaa, carries only spin, and the chargon operator, 
e 1 ia , carries only charge. The enlarged Hilbert space is 
constrained by L ia = E CT / iaa /»ao— 1, where L ia = n ia -l 
represents the charge quantum number, conjugate to 0j o . 
Integrating out Lj , appendix YK[ we obtain the partition 

function, 2 = J Df*DfD8Dhe~f dTL , where the Eu- 
clidean Lagrangian is given by 

L = Y1 fia<r d rfia<T + ^{ih la - Ha) I ^ fiaafiaa ~ 1 J 
*)0 i,a \ er / 
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-V/V- f* f e^'-^+hc) 

E^+ + ^+) 2 + E^- + ^-) 2 - ( 2 ) 

Here U± = 2U ± C/' and A± = (A x ± A 2 )/2 for A a = ia 
or /ij a , where /ii a is a Lagrange multiplier field enforc- 
ing the constraint. In this paper, we concentrate on the 
parameter region [7_ << [7+,t, in which case the phase 
stiffness for the 6i- field is large and the phases of the 



chargon fields in the two layers are locked together. If 
9i_ is condensed, both 0j_ and are gapped due to 
the Higgs mechanism. At low energies, we can set the 
chargon fields and the Lagrange multipliers in the two 
layers equal to each other and our model reduces to the 
model with one chargon field, 9i, and one Lagrange mul- 
tiplier, hi. 

We now decouple the quartic terms in the hopping sec- 
tor by a Hubbard-Stratonovich transformation to obtain 
the effective Lagrangian, 



L 



E +tLC 



<i,3> 



E *' 



Xu Xu +h.c 



E^+E A *+E /ii 



E &*@r ~ - t^f™ - E 1 [xfjfLJia. + h.c] - J2 * [xfj^ftiJj^ + h.c 



— E + ^) x * - ^) ^ - E * [4 



yXf^+h.C 



<*J> 



E * 



(3) 



Here a soft boson field X.- L = e~' t6i has been introduced 
with a Lagrange multiplier A^ which imposes the con- 
straint \Xi\ = 1. Aj = — iXi and /if = — i/i, are the saddle 
point values of the Lagrange multipliers and lie on the 



imaginary axi: 



,16 
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and x* {x{j and xf/) arc the nn 
(nnn) hopping order parameters of spinon and chargon 
respectively. 



III. MEAN-FIELD PHASE DIAGRAM 

In the small U limit, the system essentially reduces to 
a non-interacting model with no coupling between the 
layers. In this limit, the conventional QSH phase will 
be realized in the first layer where there is a spin-orbit 
coupling. In the second layer, the semi-metal (SM) phase 
with gapless Dirac fermions will be obtained. 

When U >> t,t , the Coulomb interactions are dom- 
inant and the low energy states of the system are de- 
scribed by the configurations which satisfy ^ a ni a = 2. 
To second order in t and t , the low energy effective 
Hamiltonian is obtained to be 



H » s = Tt E *[QiQi 



u 

ll 

u 



<hJ> 



E 



tr 



e -i<pi 3 cr 3 q- e i<p tj <T 3 j- 
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where [Qi] aCT)6(T < = c\ aa c iW is the 4 x 4 matrix of U(4) 
generators and [Ti} a(J > = ej la .c ilo .< is the 2x2 matrix 
of U(2) generators which are restricted to the first layer. 
The first term has a [7(4) = [7(1) ® SU{A) symmetry 
where the [7(1) is associated with conservation of the to- 
tal charge and the SU{A) with conservation of the flavor 
quantum number given by the layer index and the spin. 
The 6 states which satisfy the constraint ^ a rii a = 2 at 
each site form the rank 2 anti-symmetric representation 
of the SU(4) group. The second term breaks the [7(4) 
symmetry into SU(2) ® [7(1) 3 where the unbroken SU (2) 
symmetry is the spin-rotational symmetry in the second 
layer and the three U(l) symmetries are associated with 
charge conservation in each layer and S z conservation in 
the first layer. If t =0, each nn bond tends to form 
an £[7(4) singlet and a valence bond solid (VBS) phase 
which breaks translational symmetry is a good candi- 
date for the ground stated. The most natural pattern 
among possible VBS states in the honeycomb lattice is 
the dimerized phase where valence bonds are stronger for 
the bonds which are directed along one of the six sym- 
metry directions^. A non-zero t will enhance quantum 
fluctuations, but we expect that the fully gapped dimer- 
ized state will remain stable for a finite range of t < t. 

With the guidance of these insights, the mean field the- 
ory is carried out for the uniform and dimer ansatze. We 
solve a system of self-consistent equations at T = for 
the link order parameters, chemical potentials and La- 
grange multipliers, by requiring that the energy remains 
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FIG. 1: Phase diagram in the space of t /t and U/t in a 40 x 40 
lattice with U- = 0. The weakly interacting phase (small U) 
has Z / and the first layer forms the conventional QSH 
phase while the second layer is in the semi-metal (SM) phase 
with gapless Dirac nodes. The intermediate region has the 
fractionalized quantum spin Hall (FQSH) phase with Z — 
where chargeless spinons form the QSH phase in the first layer 
and the gapless spin liquid (SL) phase in the second layer. In 
both QSH/SM and FQSH/SL phases, the nn and nnn hop- 
ping order parameters are nonzero and site-independent. The 
large U region is a dimerized phase where the hopping order 
parameters along the bold lines have the maximum amplitude 
and all other bonds have zero amplitude. The solid line rep- 
resents the second order transition and the dotted line, the 
first order transition. 

stationary with respect to variations of those variables. 
We then find the mean-field phase diagram by choosing 
the lower energy configuration between the dimer and 
uniform ansatze. In particular, we are interested in find- 
ing a new phase in the insulating side of the phase dia- 
gram where both the spin-orbit coupling and the electron 
correlation are important. Although, we could start from 
an effective 'spin' model to study such insulating phases, 
we will use the full action in Eq. (J3]) which is applicable 
in all parameter regimes. It would be of interest to study 
the possibility of obtaining an exotic phase in an effec- 
tive model like Eq. ^ , possibly with additions of higher 
order ring-exchange terms. 

For large U, we find the dimerized configuration has 
lower energy, while for small U the uniform configura- 
tion has lower energy, as expected. There is a first or- 
der phase transition between these two phases. Within 
the uniform phase, an onset of the chargon condensation 
marks another phase transition. Although not shown 
here, the chargon gap vanishes continuously as U de- 
creases and the phase transition is a second order phase 
transition. The Bose condensation amplitude is given 
by Z = | < X > | 2 . If the chargeons are condensed, 
a spinon recombines with a chargon to become an elec- 
tron. This phase is the conventional weakly interacting 
phase where the electrons form the QSH phase in the 
first layer while the semi-metal (SM) phase with Dirac 
nodes is realized in the second layer. For t ft < 0.2, 
the first order phase transition from the uniform phase 



to the dimerized phase occurs before the Bose condensa- 
tion amplitude Z becomes zero as U/t increases so that 
there is no intermediate phase between the conventional 
QSH/SM phase and the dimerized phase. On the other 
hand, for t ft > 0.2, a window opens up for an intermedi- 
ate phase and the region of stability for the intermediate 
phase grows as t is increased. The mean-field phase di- 
agram is shown in Fig. [TJ 

The intermediate phase is characterized by the uniform 
link order parameters but, unlike the QSH/SM phase, 
the chargeons remain gapped, which makes it an insu- 
lating phase. This is a phase where the fractionalized 
spinon arises as a low energy excitation. In the first 
layer, the spinon is gapped in the bulk due to the the spin 
dependent phase in the spinon hopping which has been 
inherited from the spin-orbit coupling of electrons as is 
shown in the seventh term of Eq. © . At the mean-field 
level, which ignores the fluctuations of the order param- 
eters, the spinon spectrum is essentially the same as the 
electron spectrum in the Kane and Mele modetk The 
non-trivial topological structure in the spectrum guar- 
antees that there exist gapless edge modes in the first 
layer. Therefore, we have a fractionalized quantum spin 
Hall (FQSH) phase in which the gapless edge states are 
carried by spinons and not by electrons as in the con- 
ventional QSH phase. It is noted that the gapless edge 
mode and the FQSH state may be robust even though 
S z symmetry is broken in the first layer as will be dis- 
cussed in the next section. In the second layer, the spins 
form an algebraic spin liquid (SL)21, whose low energy 
excitations are described by four two-component Dirac 
spinons. 

The electromagnetic response and transport properties 
of the FQSH phase are very different from those of the 
usual QSH phase, as discussed below in Sec. V. It was 
recently pointed out that the conventional QSH state can 
have spin-charge separated excitations in the presence of 
7r-flux even in the absence of many-body interactions^. 
We emphasize that the spinon which arises in the FQSH 
phase is different in that they are intrinsic excitations re- 
sulting from many-body correlations while the fraction- 
alized excitations obtained in the non-interacting sys- 
tems are generated by an external fractional magnetic 
flux quantum. The QSH effect in the presence of a 
Zi gauge field was recently studied where the dynamic 
fluxon makes the fractionalized excitation a propagating 
models. 



IV. STABILITY OF THE EDGE MODES 

Beyond the mean-field approximation, the most im- 
portant fluctuations are the phase fluctuations of the 
hopping order parameters. The phase mode is described 
by a gauge field because it restores the gauge invariance 
associated with the local phase transformation fi aa — > 
e zipi fiaa and 0, — > 0, + (pi . The low energy effective the- 



4 



ory in the FQSH/SL phase is given by 
S = ^2 J drdx 1 dx 2 4>na{iY D M )VW 

71,(7 

+ J ' drdxidx 2 fnvfuu + J drdx! f](ij a D a )r){5) 

Here ijj ncr is the 2+1D massless Dirac fermion in the 
second layer, a labels the Kramers doublet which cor- 
responds to spin in the S z conserved case and n = 1,2 
is the index for the nodal points. D„ = — ia^ is the 
covariant derivative, is the internal gauge field and 
ffj, v is the field strength tensor with fj, = 0,1,2. 77 is 
the 1+1D Dirac fermion on the edge of the first layer 
with a = 0,1. The edge is assumed to be along the 
X\ direction. Although the gauge field is a compact 
U(l) gauge field, the compactness is unimportant at 
low energies when S z is conserved^ or a large number 
of gapless Dirac fermions are coupled with the gauge 
field^. In our case, there are N — 4 gapless Dirac 
fermions coming from the second layer. In the following, 
we proceed with the assumption that the four gapless 
Dirac fermions are enough to stabilize the fractionalizcd 
phase against proliferation of instantons. It is noted that 
the stability of the FQSH state relies on the existence 
of both layers. The spin dependent nnn hopping in the 
first layer opens up the gap of the spinon in the first 
layer which provides the robustness of the edge modes. 
The presence of the second layer is crucial in that the 
gapless spinons screen the gauge field and suppress the 
gauge fluctuations. 

The U(l) gauge field is coupled to the spinons in both 
layers. The spinons are gapped in the bulk of the first 
layer but there are gapless edge modes. Although the 
existence of the gapless modes has been inferred from 
the mean-field band structure which has a non-trivial 
topological order, the stability of those edge modes 
is less clear in this case because they are coupled to 
the gapless gauge field. The key question is whether 
the fluctuating gauge field destabilizes the topological 
order associated with the spinon band to open up 
a gap for the edge modes. In order to address this 
question, one can integrate out the bulk degrees of 
freedom in Eq. ([5]) to obtain an effective action at 
the edge. The resulting theory is an 1+1D quantum 
electrodynamics (QED) with a non-local action for the 
gauge field. Whatever this non-local action is, in the 
1+1D QED the quantum fluctuations of the gapless 
fermions open up a gap for the gauge field^. This 
suppresses the fluctuations of the gauge field at the 
edge although the gauge field remains gapless in the bulk. 

One may worry about the possibility of direct spin-spin 
interactions between the two layers destabilizing the edge 
modes. To examine the stability of the edge modes, one 
has to consider all the gapless modes in the low energy 
theory Since there is no tunneling between the two 
layers, the lowest order inter-layer interactions that one 



FIG. 2: (a) Transverse spin response to an applied external 
magnetic field in the conventional quantum spin Hall phase. 
Upon threading a magnetic flux quantum, a spin up propa- 
gates from one edge 1, say, to edge 2 and a spin down prop- 
agates from edge 2 to edge 1. (b) The response in the frac- 
tionalized quantum spin Hall phase. The external flux does 
not generate any transverse spin transport because the edge 
modes are charge neutral spinons. 

can add are two-body terms of the form 

V J drdxi 4)(t 1 xi 1 x 2 = 0)i/j(t,Xx,x 2 = 0)fj(r, Xi)t)(t, aq) . 

(6) 

Since the edge modes in the first layer can only interact 
locally with the bulk modes in the second layer the inte- 
gration measure only has one spatial and one temporal 
component. Neglecting gauge fluctuations and forward 
scatterings of the edge modes, the free low energy the- 
ory is invariant under a scale transformation (r, xi,x 2 ) = 
b(r , aq, x 2 ), ip = b~ x i\) and r) = b^ 1 ^ 2 !] with b > 1. The 
inter-layer interaction scales as V = b~ 1 V. If we in- 
clude gauge fluctuations and forward scatterings of the 
edge modes, the edge mode is described by the Luttingcr 
liquid with a nontrivial Luttinger parameter K ^ 1 and 
the spinons in the second layer are described by the al- 
gebraic spin liquid. As a result, the scaling dimension 
of the inter-layer coupling will receive loop corrections 
which are of the order of [V] = -1 + 0(1/N) + 0(K - 1). 
Given that N = 4, the inter-layer coupling may remain 
irrelevant if the forward scattering is sufficiently weak. 

V. PHYSICAL PROPERTIES AND DISCUSSION 

Now we discuss physical manifestations of the FQSH 
state. The longitudinal transport properties along the 
edge are very different from those of QSH states or trivial 
insulators. There will be a metallic thermal conductivity 
along the edge due to the gapless edge mode. However, 
there will be no charge conductivity because the spinon is 
charge neutral, which is the signature of the spin-charge 
separation. 

The most stark difference from the conventional QSH 
state lies in the transverse spin transport induced by an 
external electromagnetic (EM) field. We put the system 
on a cylinder with two edges at the ends of the cylin- 
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der. In the usual QSH state with S z conservation, upon 
threading a magnetic flux quantum through the halo of 
the cylinder, a spin up electron is transported from one 
edge to the other while a spin down electron is trans- 
ported in the opposite direction. This results in a trans- 
port of net spin S = 1 from one edge to the other. This is 
illustrated in Fig.^a). On the other hand, in the FQSH 
phase, the edge modes are neutral spinons which are not 
directly coupled to the external EM field and there will be 
no such transverse spin transport. Although spinons are 
indirectly coupled to external EM fields through char- 
geons, which are coupled to both the external and in- 
ternal gauge fields, the weak coupling cannot produce 
a nonzero spin Hall transport because of a non-trivial 
quantum order associated with the fractionalization. In 
the fractionalized phase, the tunneling rate of the inter- 
nal gauge flux from one value to another value is expo- 
nentially suppressed with increasing system size and the 
flux through the cylinder is precisely conserved at T = 
in the thermodynamic limit. The internal gauge flux re- 
mains strictly at zero under the adiabatic insertion of the 
flux. Therefore, the external flux does not induce any 
transverse spin transport, as is illustrated in Fig. [2Kb), 
in sharp contrast to the QSH state. This insensitivity of 
the edge modes to EM fields can potentially be useful in 
stabilizing the edge modes in an environment with fluc- 
tuating EM fields which induce back scatterings between 
the edge modes in QSH states. 

In summary, we proposed and studied a simple model 
which has both spin-orbit coupling and many-body in- 
teractions. We found a region of the mean-field phase di- 
agram where a fractionalized quantum spin hall (FQSH) 
state is stable and argued that this state may survive the 
effects of fluctuations under certain conditions. In the 
FQSH state, charge neutral spinons form gapless edge 
modes which carry only spin, unlike the conventional 
QSH state where the edge modes carry both charge and 
spin. Due to the charge neutral edge modes, the FQSH 
state shows a set of unique transport properties and elec- 
tromagnetic responses which are distinct from conven- 
tional states. 
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APPENDIX A: DERIVATION OF THE BOSON 
ACTION IN EQ. © 



where 1 and 2 refer to the layer index. Here Hl = U (L\ + 
L§) + U L\Lq, — i(h\L\ + hiL-z)- The one site partition 
function becomes 

Z g _ e ih(6[~e 1 )+il2(6' 2 -6 2 )-e[u(l 2 1 +ll)+u'lil2~i(h 1 l 1 +h 2 l2)] 

where we have omitted the site dependence of the eigen- 
values to simplify the forthcoming formulae. 

To obtain the effective action of the 9i a variables we 
must sum over all h a . We do this by making a change of 
variables from the discrete U a to a new set of 'continuous' 
variables p a — el a . After implementing these changes we 
then change variables from the original p\ and p2 to new 
symmetric and antisymmetric variables 



P± 



- (p 1 ±p 2 ); 



(A3) 



which allow us to write the one site partition function as 
two decoupled Gaussian integrals 



1 

2? 



dp+dp- 



xe 



2ip + [e + +/» + ]+2«p_[e_+fc_]-i [(2U+U')p\ + {W-U')p 2 _ 



(A4) 



where we have rewritten all fields as symmetric and anti- 
symmetric combinations of the original layer dependent 
fields. Defining new coupling constants as U± = 2U±U , 
we obtain 



1 

2^2 



(A5) 

The full partition function for the 9 variables is ob- 
tained by taking a product over all lattice sites of the 
single site result above. This gives the last two terms in 
Eq. © 



APPENDIX B: EXACTNESS OF ONE-BOSON 
THEORY WHEN u' = 2U 

In section HIl we argued that in the region U « 2C/ our 
model reduces to the one-boson model through the Higgs 
mechanism. Here we show that the one-boson model be- 
comes exact when U = 2U. For U — 2U, we can write 
Hamiltonian JTJ) as 



H — (tij a(T cl aa .Cj aa + Il.i 



Since the constraint Li a — rii a — 1 is diagonal in site 
indices calculating its partition function can be reduced 
to calculating one site matrix elements of the form 



effr 



|0102> 



(Al) 



\a=l J 
-^2 Ha (nia - 1), 



(Bl) 
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where we have introduce an SU (4) index, a = 1, .., 4, de- 
fined as 1 = (1 t), 2 = (1 4), 3 = (2 t), and 4 = (2 |); 
the first letter in the parenthesis is the layer index and 
the arrows represent the eigenvalue of S z . The Coulomb 
term is now an ST/ (4) symmetric interaction term. 

We can now decompose the electron operator into a 
spinon part and chargeon part as Ci a = fi a s~ l8i , where 
the SU (4) quantum number is carried by the spinon. 
With this decomposition we obtain the slave-rotor repre- 
sentation for an SU(4) models, 



-£M«(/Ua-^)- (B2) 

Here hi is the Lagrange multiplier which imposes the con- 
straint Li = Y, a fiafia ~ 2 with L l being the conjugate 
variable to 6*;. We have defined a new chemical potential 
M« = Mi if a = 1, 2 and jj, a = //2 if ex. = 3, 4. If we apply 
the similar Hubbard-Stratonovich transformation to this 
Hamiltonian we would reproduce the effective action in 
Eq. ©. 
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